1. Field of the Invention
The invention relates to a tester for testing an electronic system, an arrangement for testing an electronic system, a computer program product for testing an electronic system, and a method for testing an electronic system.
2. Description of the Background
The goal of protocol conformance testing is to ensure that different products, often from different vendors, are interoperable. That is, they speak the same language and can work together. Conformance testing is described in Ian Sommerville: Software Engineering (7th Edition), 2004, ISBN 0321210263. Conformance testing usually has the following steps (Sommerville, page 539):
1. Design and create test cases: A test case is made up of input and expected output, which cover the intended behavior of the tested product.
2. Run the tests and note the differences in the expected behavior and the tested product. The problem with this approach is the difficulty of coming up with correct expected outputs. It is especially hard when specifications are incomplete. The number of test cases tends to be quite low because of the effort required to design the expected outputs. Interoperability tests, where different products are run against each other, are still required since not all relevant input and output patterns are recognized in the test case design.
Model-based testing is another way to create conformance tests. Model-based testing is described in M. Blackburn, R. Busser, A. Nauman: Why Model-Based Test Automation is Different and What You Should Know to Get Started, 2004, Software Productivity Consortium. Model-based testing usually has the following steps:
1. Create a model of the tested system.
2. Run automation that creates a set of test cases from the model.
3. Run the tests and note the differences in the expected behavior and the tested product.
The expected outcome for each test case is determined by executing the model. The problem is that the effort of creating the model can be compared to creating the actual conformance test cases. Still, this method can be used to create a larger number of test cases. The accuracy and relevance of test cases depends solely on the model, which adds a level of indirection since test cases are not created directly.
Sometimes, test case creation and test run are done at the same time, so that test cases are generated and run simultaneously. The number of test cases is not pre-defined, since the responses from the tested implementation affect the upcoming test cases. This is called exploration testing, and it is described in J. Helovuo, S. Leppänen: Exploration Testing, Second International Conference on Application of Concurrency to System Design, 2001. In automated regression testing, outputs gathered when running an earlier version are used as expected outputs to a newer version (Sommerville, page 564). The purpose of regression testing is to see that the changes introduced to the newer version have not introduced any extra changes to the new version. In back-to-back testing, two implementations of the same protocol are tested by identical inputs to ensure that their behavior is identical. However, both regression testing and back-to-back testing are limited to situations where results of two implementations are compared to pinpoint differences between them.